Presenting information in a conference

ABSTRACT

One conferencing system allows conference participants to share all or a portion of the display seen on their computer screens. The conferees may be at sites removed from each other, or may view a recorded presentation or archived conference at different times. Conference participants are either “presenters” who can modify the display or “attendees” who cannot modify the display. A pointer icon, which can be labeled to identify the conferee, is displayed on the shared image area. Each conferee can modify the position of his or her own pointer, even when not presenting, so that every participant can see what each conferee is pointing to, should a conferee choose to point to an element of the display. These and other features apply to other data streams shared in the conference or in meetings where there is no shared-image data stream.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation and claims the priority benefit ofU.S. patent application Ser. No. 13/780,858 filed Feb. 28, 2013, whichis a continuation and claims the priority benefit of U.S. patentapplication Ser. No. 13/245,590 filed Sep. 26, 2011, now U.S. Pat. No.8,396,923, which is a divisional and claims the priority benefit of U.S.patent application Ser. No. 12/854,140, filed Aug. 10, 2010, now U.S.Pat. No. 8,862,972 which is a divisional and claims the priority benefitof U.S. patent application Ser. No. 11/925,960 filed Oct. 28, 2007, nowU.S. Pat. No. 7,877,489, which is a continuation and claims the prioritybenefit of U.S. patent application Ser. No. 10/753,702 filed Jan. 7,2004, now U.S. Pat. No. 7,310,675, which is a continuation and claimsthe priority benefit of U.S. patent application Ser. No. 10/600,144filed Jun. 19, 2003, now U.S. Pat. No. 7,197,535, which is continuationand claims the priority benefit of U.S. patent application Ser. No.09/523,315 filed Mar. 10, 2000, now abandoned, which is a continuationand claims the priority benefit of U.S. patent application Ser. No.08/823,744 filed Mar. 25, 1997, now U.S. Pat. No. 6,343,313, whichclaims the priority benefit of U.S. provisional application 60/014,242,filed Mar. 26, 1996, the disclosures of which are incorporated herein byreference.

This application is related to U.S. patent application Ser. No.11/086,506 filed Mar. 21, 2005 and U.S. patent application Ser. No.11/086,507 filed Mar. 21, 2005.

BACKGROUND OF THE INVENTION

The present invention relates generally to the field of shared computercommunications and computer conferencing. In particular, one embodimentof a conferencing system according to the present invention facilitatesthe conferencing of two or more persons, each with a computer at one ormore locations with a shared visual display and additional communicationcapabilities such as video, shared drawing, audio, text chat, etc., andfacilitates the recording and later playback of the communications.

Existing conferencing systems can be described as either videoconferencing systems or “whiteboard” systems. In a video conferencingsystem, a snap-shot of the conference presentation is taken at regularintervals, such as thirty times per second. Given that the image on acomputer display is not changing nearly that often, video conferencingwastes large amounts of bandwidth. In a whiteboard system, the presenterat the conference draws within a whiteboard application or imports theoutput of another program into the whiteboard program for manipulation.When the presenter is ready to present a snap-shot, the presenterpresses a “send” button and the whiteboard program updates all theattendees' displays with the image created by the presenter. This typeof system, while requiring less bandwidth than video conferencing, isclumsy to use, lacks real-time responses, and limits the presenter tothe tools provided with the whiteboard program.

Existing shared-display or shared-image systems rely on interception andcommunication of display or graphics system commands or depend onconferees' having similar hardware and software platforms. These systemslack flexibility and performance if the network connections areunreliable or have narrow bandwidth, or they require uniform hardware orsoftware installations.

Existing systems that provide single or multiple data stream handling ofa nature different than shared-image conferencing depend on widebandwidth network connections or on all participants having similarplatforms.

SUMMARY OF THE CLAIMED INVENTION

An improved general purpose data-stream computer network transportsystem and, in particular, an improved desktop conferencing system isprovided by virtue of the present invention. The desktop conferencingsystem is used to display a shared collaboration among conferenceparticipants (“conferees”), with one or more individuals located at eachremote site connected to the conference. Typically, at any particulartime some conferees are not able to modify the shared images, and thusthey are “attendees,” as opposed to “presenters.” Preferably, only oneconferee is the presenter at anyone time. A pointer icon for eachconferee can be displayed on the screen, and the conferee is able tomodify the location of his or her pointer, even if the conferee is notone who can modify the shared display itself. Each of the pointers canbe labeled to distinguish each of the conferees.

In a specific implementation of the desktop conferencing system,conferee client computers (“conferee clients”) connect to the“conference server,” a computer or several networked computers (any ofwhich may also be used by a conferee as a client computer) runningconferencing software, typically by navigating a World Wide Web (“WWW”or “Web”) browser through a predetermined Universal Resource LocatorCURL″) that indicates a Web page describing the conference. Theconference can be set up any time earlier by anyone with access to thisserver function. At the time of setup, one or more password characterstrings (“keys”) can be specified for the conference. The key that aconferee gives at the time of attempting to connect to the conferenceserver determines whether that conferee will be allowed access to theconference and what the conferee's initial privileges will be forparticipating in the conference and for modifying the setup of theconference. These privileges include but are not are not limited to thefollowing: entering the conference, being a presenter, having a pointer,seeing the icons or other identifying information of other attendees,hiding or sharing one's own icon or identifying information, changingdescriptive information such as the name, time, and purpose of theconference, changing keys, and changing others' privileges. Theprivileges can be modified during the conference by conferees or otherswho are so authorized. In general terms, the privileges include thosethat conferees might enjoy in person at a conventional, physicalmeeting. In the description below, a conferencing or othercommunications session provided by the present invention will sometimesbe called a “meeting.”

A presenter uses his or her computer to begin a conference presentationby connecting to the conference server. Conferencing software on thepresenter client computer captures a portion of the screen display ofthe presenter client and sends the captured region (after possiblycompressing it or applying other transformations) to the conferenceserver. The captured region can be anything the presenter client canhave displayed on its screen or a portion thereof, whether or not thehardware or other software producing or managing any part of the displayis aware of the conferencing system.

When the attendee selects a link from the Web page to begin theconferencing session for that attendee, this action initiates theattendee client conferencing software. The attendee client then obtainsa current view of the captured region from the conference server. Theposition of a pointer icon on a conferee's view of the captured regionand an icon specified by the conferee might be communicated to each ofthe other attendee and presenter clients, so that each of theparticipants can see what each conferee is pointing at should a confereechoose to point to an element of the shared captured region. Aparticular conference can include more than one presenter; all confereesmay be presenters, or all conferees may be non-presenting attendees. Thelatter may happen if a conference is set up to review a previouslyrecorded or archived conference.

In a simple embodiment, the entire screen of the presenter is shown toall of the attendees. In a more complex embodiment, multiple subsets ofmultiple presenters' screens might be made available to all attendeeswhile other subsets of the displays of the presenters are viewable by asubset of the attendees, thus allowing private side “conversations.”These side conversations can be flexibly reconfigured during theconference, according to the conferees' privileges; participants in sideconversations can have separate pointers whose positions are independentof, and whose labeling icons are distinguished from, those appearing inthe general conference.

As each conferee joins a conference, the client and the conferenceserver agree on the capabilities of the client, such as displaybit-depth, bandwidth of the connection between client and the conferenceserver, processor speed of the client, and the amount of memoryavailable to each client. These parameters may be modified by theconferee, the client, or the server: this can be done automatically oron demand. If the conference server determines that a client hassufficient computing resources, some of the tasks, such as image datacompression (for presenter clients), decompression (for attendeeclients), update scheduling (both types of clients), and other imagetransformations and server management functions can be assigned to theclient computers. The client computers might be personal computers,workstations, X-terminals, cable or satellite TV set-top boxes (“STBs”),personal digital assistants (“PDAs”), game playing machines, WebTV™,network computers (“NCs”), Infopads, visual telephones, and otherexisting or as yet undeveloped input and/or output devices. Theseclients might be connected to the server computer or computers (and theserver computers might be interconnected) by high or low bandwidthelectrical or optical connections, radio, infrared, microwave, telephonemodem, or hybrid combinations of these, or other existing or as yetundeveloped data communication technologies.

The system can supply a range of coder-decoder (“codec”) facilities forthe compression and decompression of images (in order to reducebandwidth requirements during network transmission) and for the matchingof image representations to client display requirements including inputor output format transcoding (in order that the shared image appearvisually similar to presenter and attendee). In addition, codecs may beprovided by the system for such purposes as error-correction,encryption, or audio and video noise reduction, or others. User-providedor proprietary codecs for these purposes and more can also beincorporated into the system. Any of these codecs may be in form ofsoftware or specialized hardware. Multiple codecs may be provided forthe same function; should the system determine that one is better suitedfor that function, then it may be selected, and the codec can be changeddynamically when conditions change, such as client requirements, serverneeds, and network loading.

At least one embodiment of the present invention provides real-time,multipoint, multi-speed transport over a computer network for datastreams other than the visual conference shared images described above,including but not limited to audio, video, shared paint and drawingspaces text chat, and other real-time object streams where intermediateupdates may be dropped; in particular, the data streams may combine anyor all of these types of data, which may be produced by multiplepresenters, and arbitrary data streams may be combined with these. Thefeatures of connecting to servers, setting up conferences, keyingprivileges, passing identifications, accommodating multiple dissimilarplatforms and network connections, and configuring subsets of confereesapply equally to these other data streams. In the more general case, the“communications server” connects the “source” and “sink” client machinesof the “communicants” during a communication session.

But the system is not limited to real-time; thus, for example, archivingis provided. It is not limited to multi-point; thus, for example, asingle user can record for later playback; being scalable means it workswell for a few users and provides a similar communications service andexperience with many users. It is not limited to multispeed; thus, forexample, data streams where lost information cannot be easily updated bylater versions can be accommodated. It is not limited to multi-stream;for the shared screen-image stream (frequently used here as an example)by itself offers great utility. Indeed, it does not require a network:for example, the same computer could be the recording and archivingserver for a presenter using it as a client; or the same computer couldrun presenter client software, attendee client software, and thecommunication server software connecting them so that a presentationmight be previewed from the attendee's point of view.

Although a simple embodiment uses a single computer as thecommunications server, a more complex embodiment connects severalcomputers in performing the server functions. The server-to-serverinterconnections can optimize routing by using information provided inthe data stream or measured on the network, optimize widearea network(WAN) usage by connecting clients to nearby servers, provide backupreliability by migrating clients, provide scalability of conference sizethrough splitting the data stream, improve performance and robustnessthrough redundant routing, and distribute functions of the system'stransport pipeline (such as compression, decompression, and updatescheduling) over several server and client computers. These services canbe provided automatically depending on resources of the computers andnetwork (for example, measured net speed and central processing unit, or“CPU,” load) and facilities available (for example, announced clientcharacteristics, such as CPU speed, compression and/or decompressionhardware, or display parameters). They can also be configured andconstrained by the server computer administrators or others withappropriate privileges.

Existing systems do not provide one or more of the following, which areexplained in greater detail below: multi-speed at server and client,multiple reconfigurable coder-decoder transformations and transcodings,storage services (for, e.g., caching, failure recovery, recording,archiving, and playback), keyed access and privilege granting, adaptableservers and clients, multiple servers, adaptive and redundantserver-to-server routing, load sharing among clients and servers,adaptive server-to-client matching, client/server and server/serverbackup and reconnection, multiple protocols for client connections,dynamic reconfiguration of server functions, and scaling beyond singleprocess, host, or network limitations automatically or upon request.

A more complete understanding of the nature, features, and advantages ofthe invention will be realized by referring to the following descriptionand claims together with the associated drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a desktop conferencing system based on thepresent invention.

FIG. 2 is a flowchart illustrating the connection of a conferee clientcomputer to a conference server shown in FIG. 1.

FIG. 3 is a block diagram of the data flow in an architecture commonlysupporting computer graphical user interfaces.

FIG. 4A is a logic diagram illustrating the comparison of new and oldcaptured images by the presenter client, when full images are compared,and the transmission of the changed information to the conferenceserver.

FIG. 4B is a logic diagram illustrating the comparison of new and oldcaptured images by the presenter client, when a new block is comparedwith the corresponding block in an old full image, and the transmissionof the changed information to the conference server.

FIG. 4C is a logic diagram illustrating the comparison of new and oldcaptured images by the presenter client, when a checksum of a new blockis compared with the checksum of an old corresponding block, and thetransmission of the changed information to the conference server.

FIG. 4D is a data flow diagram illustrating the updating of the storedold image with a new captured block by the presenter client, and thetransmission of the changed information to the conference server.

FIG. 4E is a logic diagram illustrating the updating of the stored oldimage in various formats with a new delta block by the presenter client,and the transmission of the changed information to the conferenceserver.

FIG. 5 is a state diagram illustrating the operation of the imagecapture process of the presenter client software.

FIG. 6A is a diagram showing attendee client block clipping.

FIG. 6B is a diagram showing presenter client block clipping.

FIG. 7A is a diagram illustrating the client consistency setting.

FIG. 7B is a diagram illustrating the server consistency setting.

FIG. 8A is a block data flow diagram illustrating the operation ofserver processes monitoring and filtering a single presenter data streamaccording to the present invention.

FIG. 8B is a block data flow diagram illustrating the operation ofserver processes monitoring and filtering multiple input and output datastreams according to the present invention.

FIG. 9A is a block diagram illustrating interconnections of severalcommunications servers and communicant clients in a singlecommunications session according to the present invention.

FIG. 9B is a block diagram illustrating interconnections of severalcommunications servers and communicant clients, including migrated andrecruited connections, in a single communications session according tothe present invention.

FIG. 9C is a block diagram illustrating interconnections of severalcommunications servers and communicant clients, including backupconnections for clients and servers, in a single communications sessionaccording to the present invention.

FIG. 9D is a block diagram illustrating interconnections of severalcommunications servers and communicant clients, including decompositionof transformation sequences and functional delegation, in a singlecommunications session according to the present invention.

FIG. 9E is a block diagram illustrating interconnections of severalcommunications servers and communicant clients, including distributionand parallelization of output queues and processing, in a singlecommunications session according to the present invention.

FIG. 9F is a block diagram illustrating interconnections of severalcommunications servers and communicant clients, including distributionand parallelization of output queue contents and processing, in a singlecommunications session according to the present invention.

FIG. 9G is a block diagram illustrating interconnections of severalcommunications servers and communicant clients, including multiple andredundant routing, in a single communications session according to thepresent invention.

FIG. 10A is a block diagram illustrating a multi-layered tree topologyfor connections of several communications servers with communicantclients in a single communications session according to the presentinvention.

FIG. 10B is a block diagram illustrating a single-layer tree topologyfor connections of several conference servers with communicant clientsin a single communications session according to the present invention.

FIG. 11 is a diagram of the example architecture for a single serverwith a single meeting, according to the present invention.

FIG. 12 is a diagram of the example architecture for a server withseveral meetings running on a single CPU, according to the presentinvention.

FIG. 13 is a diagram of the example architecture for a single meetingmanager directing several servers with several meetings, running onseveral CPUs, according to the present invention.

FIG. 14 is a diagram of the example architecture for several meetingmanagers directing several servers with several meetings, running onseveral CPUs, according to the present invention.

FIG. 15 is a diagram of the example architecture for several meetingmanagers directing several servers with several meetings, running on thesame CPU, according to the present invention.

FIG. 16 is a diagram of the example architecture for a single serverwith a single meeting, but the meeting is controlled by severalinstances of a communications session server CCSS″) running on the sameCPU, according to the present invention.

FIG. 17 is a diagram of the example architecture for a single meetingmanager directing several servers with a single meeting where themeeting is controlled by several instances of a CSS running on the sameCPU, with additional CSSs for the same meeting running on other CPUs,according to the present invention.

FIG. 18 is a diagram of the example architecture for several meetingmanagers directing several servers with a single meeting where themeeting is controlled by several instances of a CSS running on the sameCPU, with additional CSSs for the same meeting running on other CPUs,according to the present invention.

FIG. 19 is a diagram of the example architecture, for several meetingmanagers directing several servers with a single meeting where themeeting is controlled by several instances of a CSS running on differentCPUs, according to the present invention.

FIG. 20 is a diagram of the example architecture for several meetingmanagers directing several servers with a single meeting where themeeting is controlled by several instances of a CSS running on differentCPUs, according to the present invention. In this diagram, thepropagation topology information is shown.

FIG. 21 is a diagram of the graph of the propagation topologyinformation in FIG. 20.

FIG. 22 is a diagram of the graph of the propagation topologyinformation in FIG. 20 together with the information for an additionalpropagation topology.

FIG. 23 is a time vs. space diagram showing some typical applications ofthe present invention.

DETAILED DESCRIPTION

FIG. 1 shows a desktop conferencing system 10 according to oneembodiment of the present invention. Desktop conferencing system 10 isshown with three attendee clients 18 and one presenter client 12.Following the arrows, presenter client 12 is connected to attendeeclient 18 through a conference server 14 and data network 16. Thepresenter and attendees may also participate in an ordinary telephonecall or a conventional conference call using telephones 20 connectedthrough conference switch 22. The voice conferencing might also becarried out through an audio connection on a data network; indeed,telephone network 24 and data network 16 could be the same.

The group of users can comprise from as few as one user, who mightrecord a presentation or lecture or video-mail onto a session archive 23for later distribution; or two people who wish to share information orcollaborate on some work product; or a small group of users, as in atypical business conference call; to many tens of thousands or hundredsof thousands of users using the network as a broadcast rather thaninteractive median. In the last case, the voice conferencing might alsobe broadcast, and could involve one-way telephone conference calls,radio, multicast network audio (such as MBone), or the like.

The presenter client conferencing software, which is usually distributedtightly bound with the attendee client software to facilitate presenterhand-offs from conferee to conferee, captures information (such asimage, sound, or other output information) from a program or programsrunning on the presenter's machine and relays it to the server, asexplained in more detail below. The server relays this information toall of the attendee client computers participating in the same sessionor conference, transforming (in manners and by methods described below)the data as required. A more detailed description of the operation ofthe system by way of the example of transporting a stream ofshared-image data during a conferencing usage of the software nowfollows.

During a conferencing session, presenter client 12 takes periodic“snap-shots” of the application screen image contained within arectangular boundary determined by the presenter, breaks the screen shotinto smaller rectangular blocks, compares these blocks to informationfrom a previous screen shot. A block that has changed is passed toconference server 14 after it has undergone possibly two transformationsand received identification marking (“ID stamps”). The firsttransformation may form the difference, using a set difference,exclusive-or (XOR), or other difference method, of the new and old blockin order to extract information on the changes only. The secondtransformation may compress the block using a publicly availablecompression algorithm such as JPEG (Joint Photographic Experts Group) orPNG (Portable Network Graphics), or licensed proprietary methods ofcompression. The need for the two transformations is determined by thesystem depending on such parameters as client characteristics, serverand network loading, and user requests. The two transformations andpossibly many others may be also performed by the server, anotherclient, or another facility available on the network.

The presenter client identifies where the block is in the capturerectangle with a block-location ID stamp; it identifies the time with atime-stamp; it may also identify itself with an origin stamp, andprovide other ID stamps as needed. In order to provide synchrony in thesystem, conference server 14 can issue time synchronization signals. Theconference server may also add time-stamps on receipt of blocks, andwill need to update time-stamps when a recorded or archived conferenceis played back.

The changed blocks, however transformed, with ID stamps, are held on theconference server until they have been sent to all attendee clientcomputers 18, or it has been determined by flow control that there is nolonger a need to hold them. Flow control between presenter client 12 andserver 14 and between server 14 and attendee client 18 determines howoften the attendee client receives information updating the image; thisflow control, described in more detail below, depends on thecharacteristics and configurations of the clients, the server, and thenetwork. Attendee client 18 can also send a command to conference server14 to obtain the latest image change information.

Attendee client 18 uses whatever change information it receives toupdate its screen display of the shared image. The attendee client mayneed to decompress the changed block information and to compositedifferences with previously received image information. The reversetransformations of decompression and composition may instead beperformed by other computers.

From time to time, attendee client 18 communicates the position of theattendee pointer (if the conferee has selected this option) toconference server 14, which distributes the pointer position and achosen identifying icon to each of the other conferee clients, which maythen add a representation of the icon or other identifying label at theposition indicated by the pointer information to the shared image on theclient's display. The purpose of these pointers and labels is to allowconferees to reference particular elements of the shared image, possiblywhile describing the elements to the other conferees over the audioconference session (via a telephone conference call or an Internet audioconferencing application, for example).

FIG. 2 is a flowchart showing the process of introducing a confereeclient 17 (client 17 refers to a generic client which might also be apresenter client 12 or an attendee client 18) to a conference ongoing onserver 14, assuming that the conference setup is performed via theWWW.First.theconfereelocatesaconferencelisting. This may be done byfinding or being told a URL or using a locator service such as ULS™ orLDAP™ The conferee also specifies an icon to be used as a pointer label.Then the conferee points a WWW browser to the conference listing, wherethe server offering this listing or an associated server validates theconferee and provides information that allows the attendee clientconferencing software to start and to connect to conference server 14itself, possibly after further validation. Other information may bepassed to the conferee client at this time as well. The connection to aserver can also be accomplished in different ways, such as using storedparameters that allow meetings to be resumed after the networkconnection is temporarily broken, or using client software having ahard-coded list of meetings. Once the attendee client software isrunning, it communicates commands and pointer icon position toconference server 14, and conference server 14 supplies an initialconference image and later screen updates to client 17 (which isinitially an attendee client 18).

An attendee can become a presenter by sending the appropriateattendee-to-presenter command to conference server 14. In the simplestembodiment with a single presenter, a message is sent to the presenter'sscreen indicating that an attendee wishes to take the presenting role;if the current presenter approves, then the roles are exchanged. In morecomplex embodiments, there can be a presenter arbitration mechanism, ormultiple presenters may be allowed. The ability for a presenter or anattendee to be involved in any particular conferencing session and theassignment of privileges in the conference can be controlled byrequiring appropriate keys from the presenter and the attendees.

Referring back to FIG. 1, data network 16 can be provided by dial-upconnections, local area networks (LANs), wide area networks (WANs),internets, intranets, the global Internet, or a combination of these orany other computer data communications links. In a specific embodiment,the conferee client computers are personal computers, workstations, orother computing hardware systems, running operating systems such asMacOS™, Windows® 3.1 or 3.11, Windows® 95, Windows® NT™, Unix™, OSj2™,NExTStep™, BeOS™, JavaOS™, or the like. There is no requirement that theoperating systems or hardware of the conferee clients all be the same.

Conference server 14 matches the form of the image to the attendeeclients before sending it. Most computer screen images are representedas a bitmap of pixels whose layout is device-dependent. To be able tosend images from one device to another, a transformation or transcodingis often required. The captured image information may be transcoded intoa device-independent format for transmission and transcoded back to adevice-dependent format for each attendee's screen; if, however, the mixof attendee clients is such that a device-independent format is notneeded no conversion is done. If these or other transcoding operationsare needed, they can be carried out at the presenter client's side, theserver side, or the attendee client's side, depending on where excesscapacity or superior capability exists. The choice of device-dependentvs. device-independent bitmaps (DDB vs. DIB) is made automatically bythe server, in response to the number and type of conferee clients. Forexample, a meeting with just two conferees, each running a Windows® PCwith similar 256-color graphics configurations, may use DDBs and achievehigh performance with low overhead. However, where differentlyconfigured clients are connected in a conference, server 14 transcodesthe images to fit the attendee's screen capability.

Multiple codecs may be involved in the transcoding of screen formats aswell as other image transformations described herein. It may even happenthat different codecs will be used on different blocks in the sameimage, depending on availability of the codec's host computer, thetransformation needs, the loading on client, server, and network, orother conditions relevant to the system's performance.

Server 14 also fits the images to the attendee's CPU capability. Server14 can be a server operated by the presenter (who would then have fullcontrol over the server's resources), or it can be owned or operated byan unrelated third party or even an attendee who never presents. It ispossible to have a third party whose involvement is solely as afacilitator of conferences. Operating server 14 is simpler thanoperating a videoconference hub, since the bandwidth is much lower. Oneaspect of the present invention is the realization that real-timeconferencing can be had without resort to full-motion videotransmission, as the item being monitored, a portion of a computerdisplay, does not change as fast as a full-motion video. A similarobservation applies to many other data communications streams.

Instead of full-motion video, attendees' screens are updated as thepresenter's screen is modified, or less frequently for attendees withslow machines. The screen is modified from left-to-right within a row ofblocks, and rows are updated top-to-bottom to improve the perception oflow latency.

In some cases, server 14 might be operating without attendees. Such aconfiguration is useful where the presenter wishes to “record” a sessionfor later playback. Even a session with attendees can be recorded forlater playback, possibly including a recording of the voiceconferencing. These stored sessions might be stored in session archive23 or elsewhere. The shared image session can be synchronized with thevoice conference by using the time stamps on the block data. When therecorded session is played back, it is an example of conference server14 operating with attendees but no presenter.

The blocks may be held at the server as full images, as differences(“deltas”) from previously received full images, as deltas from previousdelta blocks, or as some combination of these, depending on thecapabilities of the presenter and attendee clients. Periodically, aserver may “checkpoint” the image deltas. To do this, the serverrequests a full image of a block from the presenter client and uses thefull image as a replacement for all the accumulated image deltas forthat block. The server might also request the entire captured regionfrom the presenter client, or send the entire region to an attendeeclient upon request.

The conference server acts as a software-controlled switch that connectsthe presenter client with the attendee clients, taking into account thatthe speed of information transfer from the presenter client can changeand the speed of transfer to the attendee clients can change and besimultaneously different for different attendees. The workload of theentire system is shared and distributed among the client and servercomputers, and even other computers that do not perform client or serverfunctions.

Presenter Client Capture Operation

The capture operation and transport technology improves over formerapproaches by reducing the amount of work required and so enhancesperformance. In addition, the technique can be tuned to best suit theworkload on the hardware and software platforms and network connectionsinvolved, manually or automatically. In particular, this tuningdynamically matches the capture operation to the amount of computerpower available (while running the other software the conferee may wishto use) and the speed of connection to the network. Existing systemsthat capture graphics display commands, transmit them, then use them torecreate the original display appear to have great compression, whichentails economy of network transmission. But comparison with thedescription below of the present invention will reveal that the savingsare not so great when the task is to communicate data streams which canbe updated by later transmissions.

The presenter selects an area of his or her computer display to beshared (“capture region”); it need not be a rectangular area. More thanone capture region may be selected at a time and multiple regions mayoverlap. The selection may be made on a screen display, in a memoryrepresentation of a display, or in an aliased representation of either;the selection can be changed at any time. If the client has multiplemonitors or multiple displays on a single monitor, independent selectioncan be made for each. A window provided by the presenter clientcomputer's operating system, or by an application or other program, maybe designated as the capture region, and then the capture region can beadjusted automatically if the window is moved or resized. This may be afixed window, or the capture operation can be set to follow theselection of the current (“top” or “focus”) window automatically. In asimple embodiment, the presenter selects a rectangular region on thescreen (“capture rectangle”). For efficient transmission, the capturerectangle is broken up into rectangular subregions (blocks) to give goodperception of response time. For example, if the presenter has selectedall of an 800-by-600-pixel screen display to be within the capturerectangle, then it might be broken up into twelve 200-by-200-pixelsquare blocks. If the capture rectangle is later adjusted smaller, theblocks are changed to be made up of smaller rectangles, or the capturerectangle is divided into fewer blocks, or both; correspondingly, if thecapture rectangle is later adjusted larger, the blocks are changed to belarger rectangles or the capture rectangle is divided into more blocks,or both. For efficient handling of blocks, the blocks are preferablykept between 1000 and 4000 pixels in size. As the blocks are updated onthe attendee's screen, they are presented from the top row to the bottomrow and from left to right within a row.

The presenter defines the shape of the capture region and can change,control, reposition, and resize it. In some computer systems, when theregion is rectangular, the capture rectangle may be marked by atransparent window that stays visible; in other systems, it isappropriate to use four graphical objects that move together to mark theboundary of the capture rectangle.

FIG. 3 shows the display architecture of a typical computer shown withapplication programs 60(a)-(c), and graphic display mechanisms 62(a)-(c)with graphics commands capture points 64(a)-(c). The graphics displaymechanisms 62 send their output to a screen image storage area 66 whichin turn presents an image to the user on a computer display 68. Existingtechniques of image sharing depend on intercepting graphics displaycommands (graphic instructions, display commands, graphics subsystemcalls, etc.) at graphics commands capture points 64 and sending thesecommands to another computer which can use them to affect its owndisplay. This appears to have an advantage in that one high-levelgraphics drawing command (e.g., “draw a blue line from coordinates (0,0)to (0,100)”) can be expressed in fewer bits of data than what would berequired to express the resulting set of pixels on the screen. In thiscase, 100 blue pixels, say using 24-bit color depth, would require 300bytes of data, compared with a graphics drawing command that mightrequire only about 12 bytes of data.

If the task to be achieved is to send a copy of this image to anothercomputer, using the smallest number of data bits, then sending thegraphics drawing commands seems, at first sight, to be a very effectiveapproach to adopt. However, two factors mitigate this apparent saving.One arises when the data stream is compressed before transmission, andthe other arises from reflection on modern graphics drawing techniques.

Compression is very effective when there is a lot of redundancy in thedata. In the example cited above, the 300 data bytes needed to representthe blue line on the image consists of a repeating set of 3 data bytes,each representing one blue pixel. This might be compress to as small as5 data bytes total, one to indicate the code, three for the color, andone to indicate the binary number of pixels. Of course, the 12-bytegraphics drawing command might also be compressible: nevertheless, theapparently huge savings ratio is not in fact realizable.

Modern graphics drawing commands include not only simple geometricdrawing operations, but also text elements with full font and spacingspecifications, and other complex constructions. This can be seen bycomparing the size of, say, a word processing document (which containsthe graphics drawing commands, font information and text) stored in acomputer file with the size of an image of that same document, say, as afax image stored as a file on the same device. A few years ago, theimage file would always be larger than the document file; now, thereverse is often true.

Furthermore, as will be seen below, reducing the amount of data by usingcompact graphics drawing commands instead of direct image data is notalways possible when applied to real-time systems where transmission oflive, changing images is required. In this case, there are two ways tomerge changes together, thus reducing the total amount of data that mustbe transmitted. The example used above applies to a single image; whenchanging images are required—as for example, with conferencingsystems—further opportunities exist to reduce the total amount of datatransmitted. This is an advantage of the present invention.

First, when graphic drawing commands update the same region of theimage, we can capture just one resulting image containing the results ofmany commands. Second, successive changes to a particular region of theimage, which will result in successive transmissions of that region, canbe composited together. The several strategies for this updating underthe present invention will now be described in more detail.

Updates for the capture rectangle may be requested by the server, orsent at fixed or variable times announced by the presenter clientautomatically or as determined by the presenter, or sent at the commandof the presenter. The blocks sent out by the presenter client are justthe blocks which have changed since the last update. From time to time,the presenter client might send the entire set of blocks. Depending onseveral factors detailed below, the presenter client might send theblocks as difference (delta) blocks as opposed to the full informationbase blocks. Base blocks are preferred where network bandwidth is freelyavailable but computing power at the client is limited.

For efficiency, the presenter client might only send out delta blocksfor areas that have changed, since delta blocks will often compresssmaller than the corresponding base block because much of the base blockmay remain unchanged. Thus, the presenter client maintains a copy of thelast capture to allow it to generate the delta blocks.

FIG. 4A illustrates this point. In that figure, the captured images usedare divided into twelve subblocks so that unchanged portions of thecaptured image can be ignored. If the block labeled “B6” is the blockbeing sent, block B6 of the current copy of the captured image 69(a) iscompared with block B6 of the most recently stored reference copy 69(b)of the captured image (the reference copy is a copy of who the capturedimage looked at some point in the past). The result of the comparisonwill determine whether block B6 has change. If it hasn't, then there isno need to transit the changes.

FIG. 4B illustrates a similar process, but there, only block is capturedfrom the current image for comparison with the stored image. Thisreduces the storage required for the comparison by nearly one half, butit limits consistency, as described later.

In FIG. 4C, the images are replaced by checksums or digests, such ascyclic redundancy check (“CRC”), DFT (discrete Fourier transform)parameters, or the results of applying hashing functions appropriate forimages, or the like. Although storage is greatly reduced, as only themuch smaller checksums need to be saved, and comparison is quick, thedigest procedure must be fast in order to provide any time economy. Themain drawback is that two different blocks may have the same checksum,and then the comparison will fail to find the difference. This can bemitigated by the choice of checksum and by the unlikelihood that thecomparison would fail twice in a row when the block changes again.

FIG. 4D shows the transmission to the server of a base block when thecomparison shows a change. The block is also sent to the stored image;this allows the stored image to be updated at the same time the changesare sent to the server.

FIG. 4E shows the corresponding situation when a delta block is sent.

Obviously many combinations can occur that would provide additionalsavings under some load conditions. Thus the stored comparison may be acollection of base blocks, either from the same capture event or not, anarray of checksums of base blocks or of delta blocks, a collection ofdelta blocks, results of compositing delta blocks, etc. or anycombination of these.

It is possible to reduce the size of the stored comparison image toblocks which have changed recently and perhaps their neighbors, as longas the full image is stored every now and then.

Each of the modes of comparison and transmission can be altereddynamically; for example, one heuristic is to send a delta block whenless than half the pixels in the capture rectangle change, and to send abase block when at least half change.

With techniques that rely on capturing graphics display commands, it isto very hard to identify commands that produce overlapping elements ofthe image. Because of this, it is hard to know when earlier commands canbe discarded because their results have been superseded. Therefore, asystem relying on capturing commands must send all commands over anerror-free network. By contrast, in this system, deltas can be droppedwithout permanent ill effects.

The foregoing assumes that the capture rectangle is broken into a numberof rectangular blocks. This decomposition can be changed dynamically tohave more or fewer blocks to adapt to changes in the size of the capturerectangle by the presenter as described earlier, or to changingconditions in the loading and capabilities of clients, servers, andnetworks. Just as the capture rectangle is broken into blocks to improveperceived rate of change, so may the block be subdivided to furtherisolate just the changed portion of the image. One way to accomplishthis is to identify the smallest bounding rectangle of the changedportion of the image, and then to intersect this with the current blockpattern. Another is to adaptively redefine the block pattern to best fitthe changed area of the image. Other adaptations arise if the geometricassumptions of rectangular capture region and rectangular blocks in thisexample are dropped.

In general, the system of the present invention is oriented to reducebuffering in order to improve the sense of “live performance.” Thus,while the structure of the server and client software could allow anumber of captured images to be in the process of traveling frompresenter clients to attendee clients at one time, in fact having justtwo images in the “pipeline” from presenter to attendee at once takesadvantage of processing capacity, defeats transient network breakdown,and does not overload end-to-end connection performance. This might beincreased to three or four or more images in the pipeline if the networkconnections are fast, but the clients have slow CPUs.

FIG. 5 is a state diagram of the presenter client software. The statetransitions are described here starting with the IDLE state. Thepresenter client is in the IDLE state while it is doing other things,such as processing data unrelated to the conferencing software orrunning another application whose output is to be shared with theattendees. Periodically, the presenter client will check to see if anupdate to the capture rectangle needs to be sent out. In consideringthat need, the presenter client conferencing software considers the CPUloading on the presenter client computer, taking into account any limitthe presenter might have placed on what percentage of his or hermachine's computing resources can be occupied with block updates, thetransmission rate of the presenter's network connection (no sensepreparing a block update if the network can't handle it), commandsconcerning flow control from the server (server flow control isdescribed below) and other relevant parameters.

If the presenter client decides to proceed, the state changes to theBLOCK-GRAB state, where the current capture rectangle or a portion of itis grabbed from display memory. A copy of the next most recent capturerectangle is maintained so that delta blocks can be easily generated. Inthis state, the delta blocks are generated if they are to be used. Ifthe delta blocks indicate that nothing has changed, the computertransitions back to the IDLE state and does not send out the capturedblock or its delta (which would be blank). Otherwise, the clientprepares the blocks which have changed for potential transmission. Thecapture rectangle is divided into blocks as described above. In theBLOCK-GRAB state, the presenter client estimates the amount of workrequired to prepare the grabbed blocks for transmission to the server,the attendee requirements, and local hardware capabilities. If thepresenter client can perform work such as transcoding much faster thanthe attendee clients, or even the server, then the presenter clientperforms that step by transitioning to the COMPRESS/TRANSCODE state. Thepresenter client might skip this state altogether if no transcoding isto be done and compression is not used (such as where the networkconnection between the presenter client and the server is much fasterthan the compression speed of the presenter client). Either way, thepresenter client then transitions to the NETWORK state, where itdetermine if the capture rectangle still needs to be sent and checkscurrent network bandwidth. Then, the presenter client transitions to theOUTPUT state where the blocks are output, either as base blocks or asdelta blocks, either compressed or uncompressed. The presenter clientthen returns to the IDLE state where the process repeats after a time.

In the case of displays that support multiple layers in applications orin the interface through multiple frame buffers or reserved areas ofmemory, the system can capture from one or more of the layers, incoordination or independently. If the client has multiple monitors, thenthe system can capture from some or all of the displays.

In general, the presenter client sends out a stream or streams, whichcan vary in format over time. The presenter client can also imbedcommand messages into a stream, such as a command indicating a changedcolor map, a pointer icon position, or a presentation hand-off command;such commands can also be sent in a separate communications channel.Capture can also occur in buffers for other purposes than screendisplay. Streams other than the shared-screen conferencing stream(outlined above and described in more detail below) can carryinformation to allow shared or broadcast text chat, audio, video,drawing, whiteboarding, and other communications. These streams aresubject to and can enjoy the same or similar load/need analysis andbalancing methods and mechanisms.

Other Client Features and Behavior

When a new conferee joins a meeting or before, the conferee selects apersonal icon and a characterizing sound (a “gong”) which will be theicon and gong that other conferees will associate with the joiningconferee. Icons and gongs can be created using well-known techniques forcreating icons and audio data. When a new conferee joins a meeting, theconferee client sends his or her personal icon and gong to each otherclient, via the conference server. The new conferee is then “announced”by the gong. The personal icon of the joining conferee is also added toa conferee icon list maintained on the server or at each client. Ifanother conferee chooses to have the icon list displayed at his or herclient, the entrance of the joining conferee can be noted when the newicon appears on the icon list. Other personal information about theconferee, such as name and electronic mail (“email”) address may beprovided by the conferee and made available to other conferees via theserver. As described earlier, the visibility of icons, audibility ofgongs, access to personal information, and so on, may be based on thekey the conferee used to enter the meeting, on the identity of theconferee (by network address or otherwise), or on a combination of theseand other validators.

The presenter can “go off-air,” i.e., suspend or pause the imagecapturing process and can “go on-air,” i.e., resume the presentation atwill. The network connections can be maintained during the off-airperiod, but no changes will be sent to the server. Similarly, anattendee can request to be off-air, and no changes will be sent orscheduled by the server during the off-air time.

If clients are so configured, conferees can be given lists or iconicrepresentations of the participants in the conference, as mentionedabove. Those conferees that are presenting, those who are off-air, andthose who are requesting to present can be marked. Various subsets ofconferees, for example those in side-conversations, those in othermeetings, those connected to a particular server, and in general thoseselected by some property of the system's current configuration, canalso be marked. The visibility of the lists and the presence of anymarkings may be controlled by users, administrators, or others, based onprivileges or other criteria. In addition, graphical representations ofa meeting or part of a meeting, or of several meetings, may be availablefor display, depending on privileges.

If the presenter client computer represents images with a varying colormap or palette, then the presenter client will send out color mapinformation when the color map changes, so that the attendees observethe same color scheme as the presenter. Color map changes can occur onthe presenter client display system as the presenter opens, makeschanges in, or closes a program, either in a window that overlaps thecapture rectangle or in a window beyond the capture rectangle used forhis or her own private work.

When a change in a block is detected, the resulting changed block (baseor delta) may be compressed, making use of any special hardware (forexample, a Digital Signal Processor (DSP), often found on MPEG boards orset-top boxes), if it is available on the client computer. Thecompression codec may be lossless, or some information may be lost incompressing and decompressing (“lossy” codec) if the particularapplication and users of the system can tolerate that and wish to takeadvantage of possibly better performance.

To ensure good usage of the network, the images are captured andcompressed before the network is available to send them if possible.Without this, the network might be under-utilized. On the other hand, ifan image is captured and compressed too early, the attendees will notsee the most up-to-date information and this will reduce the efficacy ofthe visual component of the meeting. To achieve this good balancebetween system utilization and the perceived response time as seen (andpossibly heard) by the meeting's attendees, the client software uses apipeline to ensure a flow of information is always available at thenetwork, with flow-controls to ensure that image capture and anytranscoding (including compression) are never too far ahead or behind inorder to balance the load among presenter client, attendee client, andconference server. Flow control is described in more detail below.

The number of blocks and the order of comparison and modification can beautomatically determined by the server or set by the presenter. Thus, ifconferees usually work with text reading from right to left, aright-to-left updating might be more appropriate.

The size of the window displaying the shared image on the attendeeclient need not match the size of the image sent from the presenterclient in linear measure or in pixel measure. If the window is smallerthan the image, the attendee can be given scrollbars to allow navigationaround the shared image. It is also possible to configure thetranscoding to scale the size of the image received by the attendeeclient.

The attendee client can also display the shared conference imageautomatically matched in size to the capture region set by thepresenter, if the attendee desires and the attendee client computer iscapable of such display. If the attendee client is displaying less thanall of the image, the bounds of what is being shown at the attendeeclient can be communicated to the conference server so that the servercan avoid sending blocks beyond the boundaries of the attendee's window.These blocks are not sent until scrolling requires them or they areotherwise demanded by the attendee. This point is illustrated in FIG.6A, where an original image 50 as represented in the display coordinatesof a screen 54 of attendee client 18 exceeds the size of a window 52dedicated to its display. Scrollbars are included with window 52 to aidin navigation. Blocks 56 are not transmitted from server 14 to client 18until the scrollbars are used or the window is resized to request thedisplay of more of image 50.

This “clipping” of unneeded blocks can be propagated back to thepresenter client by the server if appropriate (for example, if there isonly one attendee), so that the presenter client does not have toprocess all blocks in the capture rectangle. This is illustrated in FIG.6B, using the same attendee configuration as in FIG. 6A. In FIG. 6B, thepresenter client 12 knows from conference server 14 that the shadedblocks 56 shown at the attendee screen 54 of attendee client 18 are notdisplayed in attendee window 52, so there is no need to capture orcompare the corresponding blocks 57, marked as “do not process” in therepresentation 51 of the capture rectangle, which is displayed onpresenter client screen 55 and shown with the an overlay 53 thatcorresponds to attendee window 52.

The shared conference image, text boxes, messages, control buttons andmenus, and other graphical elements may be grouped in a single window orsplit among several windows on the client's display.

Consistency is a property of the display that can be chosen at the costof somewhat reduced perception of image update speed at the attendeeclient. Both the server and client can be constrained to be consistent;server consistency is discussed below. FIG. 7A gives a simple example ofclient consistency. An entire capture rectangle with four blocks is sentby the system, and the client waits until all four blocks are receivedbefore displaying them. Thus, the entire screen represents the samepicture to the attendee as that seen somewhat earlier by the presenter.With consistency turned off each of the four blocks is displayed as soonas possible, which leads to blocks from a previous transmission beingseen alongside newer blocks, so the screen picture, at least for a time,is not consistently one that has been viewed by the presenter.

When a presenter makes a change to the part of screen that is in thecapture rectangle, a signal can be given to the presenter client via theserver when all attendee clients have received the update that resultsfrom the change. The presenter is then assured that all other confereeshave seen the change he or she has made. An example of how this can beaccomplished is given by the following. The conference server is awareof the geometry of the capture rectangle and the blocks are constantlyscanned from left to right, starting at the top and moving toward thebottom. Thus the block in the lowest rightmost position signals the endof data from a particular rectangular capture by the presenter client.Since this block may not have changed and may never arrive in theserver's input queue, a flag may be set by the server to indicate whichblock is last when a block from a new capture arrives before the lastblock has been sent to an attendee client. If neither of these twomechanisms works, the presenter client can add a message via the serverto the attendee client stating that the rectangle has been finished.Thus the attendee client can respond when it has received the entirerectangle.

In addition, a signal can be generated by the presenter client when thepresenter has made no change for some set period of time or number ofcapture cycles. This can be relayed by the server to attendee clients,so that attendees may know that after the appropriate captured image isreceived, what they see is also representative of what the presentercurrently sees.

If the connection to the server is lost, the client can notify theconferee and may then attempt to reconnect to the conference session,using saved parameters. The reconnection may be to a different server,as described later.

In the above description, “client” has referred to a computer systemcomprising hardware, an operating system, and applications software,possibly or specifically including the software necessary to participatein a conference or communication session according to this invention.All of the described operations also apply to the case when one or moreusers are running two or more instances of the client conferencingsoftware on the same computer platform. This might occur when a userwishes to be a conferee in several different conferences simultaneously,or even multiple conferees in the same conference. For example, he orshe can be a presenter in one conference and an attendee in another. Asingle person can have different identities in each client instance, sothat John Smith may be known as “John” in one client instance and “Mr.Smith” in another. Two people might be using the same computer hardwarealternately, and both can be participating in different conferences, orin the same conference as different conferees. The capture region of apresenter in one conference may include attendee displays from another,or this “chaining” feature may be prohibited by the system.

Another example of several clients running on one CPU occurs whenseveral users are connected through terminals (e.g., X-terminals) to ahost computer which is running the client software for each user.

If a user has a multiprocessor platform with several CPUs, then thesystem's client software might be configured to use two or more CPUs forthe functions of a single client during a communications session.

Server Operation

This example of operation of the invention is based on sharing computerscreen images; other stream types may be handled in a similar mannerwith similar logic, methods, and mechanisms. This is but one possiblemethod of making the server.

At the server, a queue of data packets is maintained and is filled froman input filter and drained by output filters, one for each attendeeclient. The input filter and each output filter can run at its ownspeed. An output filter feeding a client connected over a slow networkwill not send every packet from the queue, but will skip over oldinformation. This filtering process is complex, especially when the datapackets represent changes from one image to the next (delta) which mustbe composited together in order to skip over delta blocks. This is thetechnique that allows the server to work with any speed network andmixed sped clients in the same meeting. The server data handlingprocesses are described in more detail below.

The server also handles control messages, such as a request to join ameeting or a message from a client signaling that it is attempting toreconnect to a meeting after losing its connection; reconnectionrequests can also come from other servers when multiple servers areinvolved (multiple server situations are described in more detailbelow). The server accepts connection requests and verifies that theuser of the client software is authorized to join the meeting. For eachclient connection, an icon and gong characterizing the user are sent tothe server and then to all conferees by the server. If a non-presentingattendee desires to become the presenter, the attendee client softwaresignals the server and a message or signal passed to the presenterclient is conveyed to the presenter to indicate the other conferee'sdesire.

The server accepts system information from each client connection andnotes the client's requirements (e.g., all images must be 256-colorimages) and capabilities (e.g., CPU speed, available hardware-assist forgraphics, compression, DSP, Windows® DDB available). From the systeminformation, the server assigns the client connection to an appropriate“output filter class” as explained below. During network communicationwith a client, the server may measure the network response and updatethe system information. As required, the server can move a clientconnection from class to class in response to changing networkcharacteristics so as to keep the clients in a class closely matched.

A special class of output filter sends data to another server instead ofa client. This server-to-server capability allows conferences to scaleto very large numbers of users. More importantly, it allows forintelligent distribution of work over a network of servers. Unlikelow-level data transport layers, such as packet routing using theInternet Protocol (“IP”), servers know the meaning of the data elementsthey are routing, so the routing can change based on the meaning of thedata in the message. For example, if the server knows that data is adelta of a display update and the computing effort required to receiveand process each delta is more than a particular client has, the servercan decide to not route the data packet to the client or to route thedata to the client via another computer (or another process on theserver) which will perform some of the necessary processing for theclient, in essence to “predigest” the data for a client that needs it.As an additional example, the server can read the time stamps in thedata messages, and based on the demands and resources of the clients,the network characteristics, and other information concerning thesystem, can decide to route the data by alternative or redundant routesthrough other servers.

Particulars of the Server Data Filtering Process

FIG. 5A is a block diagram showing the flow of data in the serverprocesses 100 used to intelligently filter and route one of the inputdata streams among those that the system may be transporting. Asmentioned above, these data streams can be real-time shared-imageconference data streams, other data streams which have similar transportand timing requirements, or arbitrary data streams. The example usedhere is the transport of a real-time shared-image conference datastream.

Generally, a data stream arrives at the server from a presenter clientand is routed to each of the attendee clients. The complexity of thediagram is due to the fact that the server must accommodate many clientsof differing capabilities. The data stream inputs from the presenterclient are shown on the left in the form of a queue; four types of inputstream for the example of the shared-image conference are shown, but inthe preferred embodiment only one will be active at a time. Possibledata stream outputs to attendee clients for the given input stream areshown on the right in the form of an editable queue.

The presenter client can dynamically change the format in which itprovides data, based on the presenter client computer's capabilities,backlog, local network congestion, and information provided by theserver. The data can arrive as uncompressed base blocks (raw data) onthe stream labeled “ubase” if the presenter client decides not to sendthe differences and decides not to compress the data. If the presenterclient decides, based on performance, network bandwidth, etc., tocompress the data, it sends the data steam as compressed base blocks(“cbase”). The presenter client can also send the data stream asuncompressed difference (delta) blocks (“udiff”) or as compressed deltablocks (“cdiff”).

As each data block is received, it is time stamped by a time stamper 102with either the true time of arrival or a later time of handling (usedfor when the conference is not a live conference, but is being playedback). Time stamper 102 may also simply validate a time of sendingstamped by the presenter client.

The data block is then fed to a server queue for that type of datablock. The server queues are labeled “qubase” (for uncompressed basedata), “qcbase” (for compressed base data), “qudiff” (for uncompresseddelta data) and “qcdiff” (for compressed delta data). Since thepresenter client provides data in only one data type (although it couldprovide all four data types, if the presenter had a fast machine, theserver was a slow machine and the network between them had excesscapacity), and the data type sent can change from time to time, filter100 uses a queue filler 104 to fill all four queues using just the onedata type provided by the presenter client. Of course, if filter 100notes that none of the attendee clients need a particular data type,that data type can be ignored and its queue eliminated.

As shown in FIG. 8A, the data type which is received can just be routeddirectly to the queue for that data type. If the received data type isuncompressed, the corresponding compressed queue is filled by runningthe received data blocks through a compressor 106 b (base data) or 106 d(delta data). If the received data type is compressed, the correspondinguncompressed queue is filled by running the received data blocks througha decompressor 108 b or 108 d. If the received data type is base data,delta blocks are generated by a delta block generator 110, which recordsa previous base block and differences it with a current base block; itmay also reference delta blocks that it creates and stores. Delta blockgenerator 110 is coupled to the ubase stream after uncompressor 108 b sothat delta block generator 110 receives the base data whether it is sentas ubase data or cbase data. Likewise, delta block generator 110 iscoupled to the udiff stream before compressor 106 d so that both qudiffand qcdiff receive the benefit of delta block generator 110.

For processing in the other direction, i.e., filling the base dataqueues having only delta data, queue filler 104 includes a compositor112. Compositor 112 gets its inputs from a base image frame store 114,the udiff stream and the cdiff steam (after being uncompressed byuncompressor 108 d or a separate uncompressor 116). Base image framestore 114 maintains the equivalent of the previous full base frame. Asdelta frames are received, they are differenced (or, more precisely,“undifferenced”) with the contents of base frame image store 114 togenerate uncompressed base data. Because the output of compositor 112 iscoupled to the ubase stream prior to compressor 106 b, the base framesoutput by compositor 112 can be used to fill the qcbase queue as well asthe qubase queue. If the presenter client can switch from base datastreaming to delta data steaming without sending an initial snapshotframe as delta data, compositor 112 should be coupled to the ubasestream and the cbase stream (or the output of uncompressor 108 b). Ofcourse, the delta queues might contain base data from time to time, suchas when a “checkpoint” is done to prevent an error in delta data frombeing propagated indefinitely.

The data blocks in the (up to) four queues are stored in time stamporder, so they may be viewed as a single complex queue of a data typecomprising multiple parallel block entries compressed or uncompressed,differenced or base. The output of this complex queue can be sent toattendee clients as is, but filter 100 includes several other outputmechanisms to accommodate disparate attendee client types. Base frameimage store 114 might also be a source of server output, such as when aclient requests a full capture rectangle image. This might occur when anattendee client has lost its place, lost its network connection andreconnected, or is joining a conference for the first time.

Although the four output queues can be viewed as a single queue, orderedby the time stamps, there could be up to four different queue entriesfor each time-stamp value. A queue synchronizer 130 uses arbitration orprioritization techniques to settle any discrepancy between presenterclient time stamps and server receipt time stamps.

The attendee clients are classified into one of three classes: Class 1clients are fast clients on a fast network; Class 2 clients are slowclients on a fast network; Class 3 clients are clients on slow networksand/or slow clients which cannot process and/or receive the datarequired of Class 2 clients. Each attendee client is assigned to aclass, on the basis of announced or measured characteristics of theclient and its network connection. Reassignment can occur dynamically asthe connection or client loading change, or when requested by theclient. A monitor process (not shown) on the server monitors theactivity of the output filters to shift attendee clients from class toclass if the clients are either too fast or too slow for the class theyare in. This is done dynamically, and the characteristics of all clientsin a class as well as those in other classes are considered in balancingall classes.

Typically, Class 1 is used for fast attendee clients on a fast network.A Class 1 client receives all the data blocks, from one or more of theserver queues. Because a Class 1 client receives all the blocks, theserver need not track which blocks were sent to which clients. SomeClass 1 clients will be able to decompress compressed base blocks asfast as they get them, and will take the blocks from the qcbase queue.Other Class 1 clients will be able to handle every data block, but onlyif they are delta blocks.

Class 2 is used for fast network connections to slow machines, such as386 s connected to corporate LANs. A Class 2 client might not be able toprocess each block, even uncompressed blocks, in which case filter 100will discard blocks. A block discarder 132 is provided in filter 100 totrack which blocks have been discarded for which Class 2 clients. Class2 clients are provided with base data types (ubase and cbase) and notthe delta data types so that block discarder 132 can discard some blockswithout any loss of critical information. There is a loss in frame ratewhen blocks are discarded, but that loss is not as critical as the lotsof delta data blocks. In addition, the memory and time required to trackdropping base blocks for each Class 2 client is much less than fortracking the discarding of delta blocks. To avoid slowing Class 2service for all Class 2 clients to the speed of the slowest Class 2client, one output of block discarder 132 is provided for each Class 2client. Thus, a faster Class 2 client will experience a higher framerate as it will receive more data blocks than a slower Class 2 client.

Class 3 is typically used for clients that cannot even handle delta datablocks on a regular basis because of network limitations or clientprocessing limitations. Even though a client might be fast enough tohandle uncompressing data blocks, if its network connection is not fastenough to send even the compressed data blocks, the client will beclassed as a Class 3 client because not even every delta block can besent to the client. So that the client can present a conference insubstantially real-time if needed, delta blocks are composited by filter100 so that multiple base and delta blocks can be, in effect, replacedby a single data block. To accommodate the differing needs of each Class3 client, a separate queue is setup by filter 100 for each Class 3client. As should be apparent, filter 100 has to do more work for aClass 3 client than for a Class 1 client, so it is to the server'sbenefit to upgrade clients as their speeds increase or their networkconnections improve.

Filter 100 maintains a “qmulti” queue for each Class 3 client. Threeqmulti queues for three Class 3 clients are shown in FIG. 5A. A qmultiqueue receives inputs from the qubase and qudiff queues. Where those twoqueues are not used, the qcbase and qcdiff queues are used instead, butare first uncompressed using uncompressors 134 b, 134 d. The delta datablocks and a base block stored in a base image frame store 136 arecomposited by a base compositor 138 to form one composited base imagefrom a base image and one or more delta images. A delta compositor 140is used to form one composited delta image from a plurality of deltaimages. The output of base compositor 138 and delta compositor 140 arethen fed through respective compressors 142 b, 142 d, resulting in fouroutput data streams (ubase, cbase, udiff, cdiff) fed to a discarder 144which discards data blocks which the attendee client for that qmultiqueue cannot handle. If the particular attendee client does not need allfour outputs (typically, anyone client will use only one output), theprocessing for those unused queues within the qmulti queue can beskipped, since the queue only needs to service that client. As with baseframe image store 114, base frame image store 136 can supply Class 3clients with full frames upon request or as needed.

Discarder 144 drops blocks based on parameters about the network andclient known to the server and to filter 100 as well as parameters andrequests (e.g., “slow down,” “speed up,” or frame rate specifications)received from the client. The dropping of blocks is preferably done on ablock-by-block basis, but it can also involve discarding all the blocksin the presenter client's capture rectangle; the related issue ofconsistency is discussed below. If it turns out that more than one Class3 client has the same requirements, all but one of the qmulti queuesmight be virtual queues. In effect, the processing for all the qmultiqueues for those similar clients is done once, with each getting a copyof the results of that processing. For example, one multi-client qmultiqueue might be handling a plurality of 386-class client machines runningover a corporate ISDN line. Other qmulti queues might then supply othersimilar machines which are connected to the server by modem (e.g., 14.4or 28.8 kilobyte data rates), LANs, T1lines, etc. If any of these lumpedClass 3 clients deviates from the common requirement, then its virtualqmulti queue would then become a real qmulti queue and would performprocessing separate from the other queues. Among all Class 3 outputqueues, the various separate compositors 138, 140 may have differentworkloads from the fact that the number of delta blocks compositedtogether and the number of blocks discarded will vary according to thecapacity of the attendee clients serviced by the qmulti queue.

The use of more than one output class avoids a slow connection'sretarding a fast connection. Filter 100 includes a buffer reclaimer 150which examines the queues to determine if portions of the queue buffershave already been read by Class1, Class 2, and fast Class 3 clients, andare not going to be used by slow Class 3 clients (they will bediscarded). If that is the case, then buffer reclaimer 150 marks thoselocations in the queue buffers as reusable, to save on memory space.

The different output classes and the monitor processes on each datastream allow the server to handle data streams at different speeds forclients of different capabilities and network connections of differentbandwidths. The streaming of update information formed into blocksimproves the perception of low latency, but it may be desirable for someapplications to reduce the mixture of blocks from different captureevents that show on the attendee's screen at one time. The system can beset to provide this consistency by delaying the updating until a wholerectangle can be shown. One form of this adaptation can occur at theserver, as shown by the example of FIG. 7B. There, a capture rectangleis broken into four blocks (1,2,3,4). The server maintains a consistencyflag which can be either “off” or “on.” If the consistency flag is offand the server receives data representing blocks 1A, 2A, 3A and 4A(taken at time A) from the presenter client and is able to send outblocks 1A and 2A, and in the meantime receives blocks 1B, 2B, 3B and 4B,the server will send out 3B as the next block, reasoning that 3B is moreupdated than 3A so it is a better block to send out. However, if theconsistency flag is on, the server sends the old blocks 3A and 4Aanyway, so that the client can maintain a time-consistent display.Following 3A and 4A, the server sends blocks 1B-4B and so on. Clearly,consistency requires additional memory and produces added latency. Thistrade-off is decided upon between the server and the receiving clients,based on a variety of factors described above. If the network, theserver and the client can easily handle consistency, then theconsistency flag might be turned on, but where display updating happensquickly and there are other constraints on the system, the consistencyflag might be turned off.

As described above, the server provides control of information flow tokeep fast attendee clients supplied with updates as often as possible,and to avoid sending slow clients updates they cannot use or that willoverburden their network connections. The server also provides flowcontrol for presenter clients, as needed, by determining the fastestrate of updating required by attendee clients, and then signaling thepresenter client to grab blocks no faster than the fastest consumer candemand them, so that the presenter will not have to waste resourcescollecting and processing data that no client can use or no network canafford to carry.

Storage Services

FIG. 8B illustrates a more complex conference server which handles themore general case. The server in the general case might maintainadditional output and additional input queue components for transmittinginformation to other servers and for storage services, includingcaching, short-term storing, recording, and archiving, and for laterplayback. These purposes are distinguished as follows: caching providesfast memory hardware support in improving the performance of the server;short-term storage provides backup and refresh capability for extremelyslow or temporarily disconnected clients, for newly connected serversthat may need information older than that normally held in the outputqueue, for quick-turnaround failure recovery, and for other short-termneeds; conference sessions are recorded when they are primarily intendedfor later viewing by users of the system who mayor may not beparticipating in the session; an archival session captures all or partof a meeting as it occurs and is intended for users who typically wereconferees in that session and have a reason to review the session later.Uses of recorded sessions, especially when they incorporate synchronizedvoice, include live online training sessions that also serve for futureoffline training, technical and marketing demonstrations, and formalpresentations that can be broadcast or accessed remotely at will.Archived sessions have uses other than review, including briefingabsentees, capturing interactions involving or aiding technical support,evaluating sales personnel, and the like. Of course, these needs andcharacterizations are not exclusive or exhaustive.

Possible features and methods for storage handling will now be listed.The emphasis will be on recording and archiving, but shorter termstorage modes will share many of these characteristics.

During any session, there can be multiple “storage server” queues, or“storage streams,” saving output to one or more media. These can becontrolled by the server itself, by recorder-like interfaces (similar toa video cassette recorder, or “VCR”) at the clients, or by otherinterfaces operated by conferees. Each stream can be independentlycontrolled, or one controller can control multiple storage streams. Thestorage facility can operate concurrently in an ongoing meeting torecord a live conference, or it can be used by itself to capture arecording for later replay.

It is possible to control who can record a meeting, how much data theycan record (by time or by disk capacity, for example); the type ofinformation they can record (by stream, by user sets, or the like), thestorage medium (disk, tape, etc.), and when recording starts. Recordingmight be set to automatically start when a certain user connects, whenthe first connection is made, when a certain number of conferees areconnected, when the first person presents, when a particular personpresents, at a particular (real) time, after a particular duration fromthe beginning of the meeting, or because of some other triggering event.It is also possible to control the end of recording, based on similartriggering events or triggers related to capacity, elapsed time, etc.The controlling can be done by a conferee at a particular client, by amoderator, or the like.

Possible storage targets include local disk files, local databaseservers or back-ends, remote database servers or back-ends, remotestorage engines relying on the data structures, controls, and methods ofthe system of the present invention (example system architectures aredescribed below), and local or remote permanent storage media (optical,magnetic, magneto-optical, etc.). Permanent storage can also be used bythe system to assist recovery from disaster. The storage stream couldalso be directed to an email message or to another computer applicationwithin the system of the present invention or beyond it.

It is possible to control the quality of storage input and playbackoutput. Each storage stream can have an associated quality parameterassociated with it so that it behaves as though connected at aparticular network speed. Thus a stream might be stored or a playbackstream might be produced that was suitable only for replay at a givenspeed. Or several playback streams could be simultaneously produced fromthe same stored information for several different particular playbackrates. If most or all of the original session data is stored, thenreplay might perform the same adaptive filtering described in FIG. 8Afor real-time “live” meetings, so that the single storage source couldbe played back at multiple, adaptive rates.

Since there would be added value in being able to access recordedinformation, it is appropriate to describe how billing controls might beincorporated. Billing could be performed when the original recording ismade, or when a recording is played back. Billing might be based onunits of time used or on units of storage consumed, at the time ofrecording or at the time of replay. Billing for recording and playingmay be independent. Any tracking that needs to be made to implement thebilling functions can be incorporated into the storage and playbackservices of the communications system.

It is possible to tailor the data stored. Since a conference typicallyinvolves multiple data streams, one or more may be chosen for storing.Some streams might go to one storage device or modality as describedabove, others might go to different ones. Synchronization betweenstreams (e.g., voice and imagery) can be maintained, even when thestreams are stored in different places and ways.

Stored information can be replayed through another communicationssession established by the system according to the present invention, orit can be sent through other communications channels, for example,email, file transfer protocol (“FTP”), or physical media transfer bypostal or courier services, etc., and replayed by the recipient usingthe client software according to the present invention. Stored materialmight be replayed from a copy local to the user's computer, or it mightbe retrieved after WWW navigation to a replay-enabled Web site.Retrieval might involve streaming the data in the ways described above,or transferring the data by email, FTP, WWW download, or the like. WithWeb based retrieval, support could be provided for browsing by content,searching by user-defined keys, controlling access by user-providedkeys, access lists, privilege levels, or user-provided payment options(e.g., credit card number on file).

Control modes on replay might include: control by server without userinteraction for a single data stream (like a pay-per-view movie in whicheach attendee who joins gets to see the playback forward from the fromthe point of joining); control by the server, without interaction butwith multiple streams (e.g., all attendees get to see the movie from thebeginning regardless of when they join the show); by an externalmoderator; by VCR-style controls at one or more client computers. Eachset of controls can affect either all the sets of streams or aparticular grouping of streams. Replay can occur at the originalreal-time recording rate, at faster-than-realtime (like fast forwardplay on VCRs), in VCR-style single stepping modes, and in the variousreverse modes as well. Random access and jump playback by index marking,by time codes, by presenter, or by other organization could besupported.

In addition to the stored meeting contents, any other document or dataobject might be uploaded and stored with the meeting (e.g., meetingagenda, minutes of a previous meeting, or supporting materials). Uploadis another type of data stream that passes into the system server and isthen relayed to a suitable storage entity residing on the same or adifferent host. Attachments can be retrieved either with specializedfunctions of the client software, by navigating Web pages and using aWeb browser, or by other retrieval mechanism. Attachments are subject toall of the same controls as the recorded meeting contents with regard toaccess, billing, playback, etc.

The above-described elements of the more generalized conference serverconcepts are illustrated in FIG. 5B. In addition to the instances of thesimple output filter processes 100, the more complex server functionsshown in FIG. 8B includes inputs for different sources, such as otherservers (where the complex server shown might be an intermediate serverfor a large broadcast), storage sites for replay and import channels,and outputs to other intermediate servers, storage sites, and exportchannels.

Multiple Servers

Up to this point, the conference server has generally been referred toas a single computer running conferencing software. The server functionsdescribed so far may be performed on several different computers runningconference server software connected over a computer network. FIG. 9Ashows a configuration with four conference servers 14(a)-(d), onepresenter client 12, and eleven attendee clients 18 (some of which areseparately identified with letters). Three conferee clients areconnected to each server. The four servers are completely connected,that is, a connection is shown between each pair of servers. With manyservers, this degree of interconnection would be unrealisticallycomplex, expensive, unneeded, and performance degrading. One of manyuseful techniques, a “tree topology,” for interconnecting numerousservers is described below.

There are three classes of advantages from having several servers activein a given conference.

Static advantages result from a configuration and division of tasks thatmay persist throughout the conferencing session. The following are amongthese advantages.

(WAN economy and local performance) A conferee may find economy in beingable to connect to a nearby server—where nearby may mean geographiccloseness, or in the same network service area, or on the same localarea network, or the like. Thus in FIG. 9A, attendee client 18(a) mayfind it cheaper to connect to server 14(a) than to server 14(c), whilein turn client 18(c) may find it cheaper to connect to server 14(c) thanto server 14(a). At the same time, there may be better performance ofthe system with these local connections compared with a longer path withmany hops to a more distant server.

(Client migration and homogeneous concentration) The advantage of havingall of a server's attendee clients be the same and additionally ofhaving them be the same as the presenter client has been discussed.There can be an advantage then of assigning similar clients to a singleserver when several servers are available and performance is nototherwise degraded. For example, in FIG. 9B, attendee clients 18(c) and18(d) are identically configured computers running the same operatingsystems with the same display configurations as the presenter client 12,so both have been moved from their original servers (indicated by dottedarrows) and reassigned to server 14(a), as designated by the dashedarrows. The same advantage may also be found when clients of a serverare in the same output class (as discussed above under the singleserver); thus, reassignment of clients in a given class so that one ormore servers have all or most of their clients in that class can improveperformance by making those servers' processing loads more uniform.Finally, as described under the discussion of WAN economy and FIG. 9A,homogeneity may also involve nearness, and for this reason clientreassignment may achieve that goal as well. In addition to reassigningclients to servers already participating in the conferencing session asabove, it is also possible for the system to recruit additional serverswhere these resources are provided but not yet assigned to theparticular meeting. Thus, server 14(b) may be automatically connected tothe server-server structure for the meeting pictured in FIG. 9B in orderto provide connections for the three clients 18(b).

(Tree branching for load reduction and scalability) In FIG. 10A, a treetopological configuration can provide economy in traffic handling andimproved performance. Information from presenter client 12 iscommunicated to server 14(a) and then to the other servers and on toattendee clients 18, following the solid arrows. In this configuration,each server is shown handling four data connections of a single streamtype; a single server would have to handle twenty-eight data connectionsto connect the presenter client to all the attendee clients. If attendeeclient 18(a) issues a command or request to server 14(a), represented bythe dotted arrow, the message will be responded to by server 14(c) orpassed to server 14(b), and handled there or in turn passed to server14(a), with these two paths also shown with dotted arrows. This meansalthough some commands or requests may need to be seen by three servers,each server will see and process only a fraction of the total suchmessages. Just as the tree configuration allows a given number ofconferees to hold a meeting more efficiently than with a singleconference server, it also means that relatively few extra servers needto be added to expand the meeting and maintain the tree configuration.For example, if R+1 is the number of data connections per server, andceiling(x) is the smallest integer greater than or equal to a realnumber x, then the number of servers S is required to hold a conferencewith C conferees, assuming one presenter and using the treeconfiguration, is (R ceiling(log.sub.R (C−1))−I)/(R−1). In particular,using R=3, which is the value in FIG. 10A, forty servers will sufficefor eighty-two clients. More realistically, if R=100, then 10,101servers will provide the advantages of the tree configuration to apresenter and one million attendees. This takes only 101 extra serversover what would be required if the presenter client were directlyconnected to 10,000 servers, each of which served 100 clients. But thelatter configuration, exemplified by FIG. 10B with R=3 and C=28, is notrealistic, since presenter client 12 would be deluged with independentcommands or requests to update, or resend, or similar messages; in otherwords, the presenter client would have traffic in excess of the capacityof any server 14. Based on distributing the server functions over manymachines, and employing this tree topology for propagating informationamong servers, server-to-server communications and management providedby the present invention allow the number of participants in a meetingto increase exponentially with only linear degradation of theperformance. Similar analysis applies if server capacities are notuniform, that is, if different servers can handle different numbers ofdata connections.

Adaptive advantages result from reconfiguration and redistribution oftasks in response to relatively long-term changes in the system duringthe conference session.

(Backup server) If a server fails or becomes isolated from the network,then its clients may be connected over previously inactive backup linksto other servers. Attempts can be made to reestablish communication witha server that has dropped out. If unsuccessful, its workload may bedistributed to other servers. In FIG. 9C, attendee client 18(c) hasconference server 14(c) as its principal server, but the dashed arrowindicates the assignment of server 14(a) as a backup server. Should theconnection between client 18(c) and server 14(c) fail, as indicated bythe “X” on the arrow between them, or should server 14(c) fail or becomeisolated from the net, then server 14(a) can respond to commands fromand provide updates to client 18(c). Presenter clients and serversthemselves can be assigned backup servers as well. Thus the dashed arrowbetween servers 14(a) and 14(d) indicates that each has been assigned asa backup for the other. Should the link between servers 14(a) and 14(b)fail, as indicated by the “X” on the arrow between them, or server 14(b)fail or become isolated from the net, then server 14(d) can take overtraffic previously routed to server 14(a). It is also possible to haveservers ready, but not active, as backups, or to have mirroring serversfor even more secure redundancy. Since the state of the conference canbe announced to all servers, the system may be configured so that adisrupted conference session can be robustly resumed with minimal lossof data and time.

(Transformation factoring) The transformations or transcodings that ablock may undergo in transit from the presenter client to an attendeeclient may include differencing, error-correction encoding or decoding(“source coding”), compression or decompression (both for “channelcoding”, or just one for purposes of bandwidth matching; lossy orlossless), encryption or decryption (“privacy, security, orauthentication coding”), compositing with other differences or with baseblocks, conversion from DDB to DIB and back, storing, replaying,copying, or the like. Editing, mixing data from different sources orpresenters, mixing data of different kinds, duplicating, changing theorder, the format, the storage or playback quality, etc., are otherexamples of transformations, which are neither exclusive nor exhaustive.Some or all of these may be performed and the order of performing themmay change. As previously discussed, some may be performed by a confereeclient, some by a conference server. When several servers are availableor when several clients have different capabilities and resources, thesefunctions may be delegated or migrated to different machines. Forexample, in FIG. 9D presenter client 12 is differencing, conferenceserver 14(a) is compressing, server 14(b) is distributing thedecompression task to attendee client 18(b) (which has decompressionhardware), server 14(d) is compositing the resulting delta block with aprevious delta block, and attendee client 18(d) is compositing theresult with an old base block. This advantage is viewed as adaptive,since the loading configuration that makes a particular distributionfavorable may change slowly during the conference session, but it couldbe a static advantage when some machines have much greater capabilitiesthan others (such as compression or decompression hardware).

(Distributed and redundant flow control) The architecture and logic ofthe filter process as described above and illustrated in FIGS. 8A,B maybe distributed among several servers and even clients. Thus, like thefunctional transformations, portions of the queues themselves, as wellas the internal operations of the filtering process, may be found ondifferent platforms at different localities in the network, and atdifferent times. Not only may the information and functionality be sodistributed to improve memory economy or gain memory or processingspeed, but the system can be made more robust by redundant storage, andmore responsive by parallelizing the pipeline. The queues may also besegmented sequentially over several platforms. These different aspectsare shown in FIG. 9E. Here, qcdiff is stored redundantly on servers14(a) and 14(b); on server 14(a), qcdiff uses the special compressionfacilities provided by an attached hardware device 15. One card ofqmulti is stored on client 18(c) (perhaps a machine with very fastreliable surplus memory) while the rest are on server 14(c). Whileqcbase is housed on server 14(b) using a compression codec (possiblyhardware based) on client 18(b), it operates in parallel with qubase onserver 14(d), which uses a discarder on client 18(d) (there isadditional undiagrammed parallellism implicit in having portions of theoutput queue on all four servers). Finally, qudiff is segmentedsequentially with the first part (qudiff.beg) on server 14(a) and thelast part (qdiff.end) on server 14(d). Note that in the last case, thetwo segments of qudiff are not even adjacent in the network linkageshown. Another type of distribution of the queues is given in FIG. 9F.Assuming the presenter client breaks the capture rectangle into fourblocks B1, B2, B3, B4, it illustrates, using just qubase, how the streamdata can be decomposed and distributed over all four conference servers14(a-d), so that the subqueue qubase.B1 of uncompressed blocks B1 are onserver 14(a), the subqueue qubase.B2 of uncompressed blocks B2 are onserver 14(b), etc.; this represents another form of parallellism. Theseare simple examples; there are more complex analogues when there aremore servers, and all of them may occur in various combinations. Thevarious techniques of RAID (Redundant Array of Inexpensive Disks)striping with recovery from errors are also applicable. All of thedistribution schemes mentioned here may also vary over time.

These are specifically adaptive advantages, but the static advantagesalso have parallels here, since a backup server may also be close, sincea change in presenter may warrant a new tree configuration, and since anoutput class change may warrant a new homogeneous concentrationreassignment.

Dynamic advantages result from reconfiguration and redistribution oftasks in response to relatively short term changes in the system duringthe conferencing session. The following are among the dynamicadvantages.

(Content-based routing) Unlike IP routing for example, the system hasaccess to the contents of the information being routed. Thus it can readthe time stamps, type of data, and other information included in thebase or delta block data or to other system data. It can use thistogether with measured properties of the network interconnections of theservers and clients to determine best-estimate optimal routing betweenand through its components. In FIG. 9G, one route from presenter client12 to attendee client 18(d) (through server 14(b)) is shown in doublearrows, another (through server 14(d)) in heavy arrows, to illustratethat one route may be preferable under some conditions, but asconditions change, the system may select a different route.

(Redundant routing) The system can send image or other data by severalroutes at once. This can improve performance, since the earliest toarrive at the destination may trigger the discard of later-arrivingdata. This can improve the resiliency and robustness of the system,since it is more likely some data will get to the destination. It canalso improve reliability or accuracy, since several versions may becompared at the destination to see if they are identical. In case ofdiscrepant data at the destination, retransmission or some arbitrationmethod can be requested, depending on the purpose of the redundantattempts to insure delivery. For example, again in FIG. 9G, informationfrom presenter client 12 may be sent by conference server 14(a) usingboth routes, indicated by the double arrows and the heavy arrows, toserver 14(d) and then to attendee client 18(d).

These are specifically dynamic advantages, but the static and adaptiveadvantages also have parallels here as well. This can be summarized inan additional dynamic advantage.

(Dynamic reconfiguration) Any of the configurations described above andthe parameters determining them and the routing schemes can be altereddepending on changes in client, server, and network capabilities, needs,resources, and loads as announced or demanded by clients, or as measuredby the system, or as specified by conferees or system administrators, orother prevailing or desired conditions.

Any combination of advantages from these three groups may apply. Therewill in general be tradeoffs among these advantages. The system can begiven specific configuration preferences, or it can automatically adjustduring use according to preset optimization goals, or it can adaptivelyset optimization goals and adjust the configuration to approach them.

Example of Server Architecture

So far, a server or each of a set of servers operating together has beenviewed as a computer performing the server functions described. Anexample of server architecture and use will now be given, withoutsuggesting the necessity for, or the exclusiveness of, this architectureto accomplish the communications serving functions on single or multipleand interconnected servers described above. Also, the previous exampleshave dealt with a single conferencing meeting or other communicationssession; the method to be described below can also accommodate severalmeetings on the same underlying hardware and the conferencing softwareas provided by the present invention. Again, the description of thismethod is not intended to suggest that this is the only way in which theinvention can accomplish multiple simultaneous communications sessions.Any references to the image-sharing example should be extended toarbitrary data steams.

FIG. 11 shows an architecture of a single server and a single meeting.The primary component of this architecture is a server manager 36(identified in this diagram as “ServMgr ‘InfoPass’”), which is directedby a meeting manager 32 (identified in this diagram as “MeetMgr‘TheCompany’”). Meeting manager 32 is an unowned, quiescent, resident,interrupt-driven process (similar to a “daemon” process used with Unixand other operating systems). It mayor may not be running on the sameCPU as WWW server 30(a); it mayor may not be running on the same CPU 38(called here “Beowulf”) as server manager 36 “InfoPass.” The servermanager is also an unowned, quiescent, resident, interrupt-drivenprocess. Each CPU that is involved in the system for providing serverfunctions in meetings set up by a meeting manager has exactly one servermanager running on it; that server manager can be viewed as the meetingmanager's agent on that Cpu. The meeting is directly supervised bycommunications session server(“CSS”) 40(a), called here “Meeting #1‘Product Support.’ II When server manager 36 receives a command frommeeting manager 32 that includes the information on a meeting and on thefirst conferee that wishes to connect, the server manager creates a CSSto handle the meeting. The CSS is an owned, evanescent, quiescent,interrupt-driven process. The CSS is owned by the server manager and iskilled a period of time after all the conferee clients connected to itsmeeting disconnect or fail to respond.

In FIG. 11 there are three conferee clients 17(a)-(c) connected to themeeting. Clients 17(a),(b) use a client-server protocol provided by thesystem, which might be a combination of Transmission Control Protocol(“TCP”) and User Datagram Protocol (“UDP”), for example. Client 17(c)uses another protocol, here exemplified as Hypertext Transfer Protocol(“HTTP”). The CSS 40(a) provides an included “gateway” layer 40(b) foreach connection protocol other than the system protocol, and this layertranslates the client's nonsystem protocol to the system protocol. Theacceptance of different protocols may aid the system's operation acrossfirewalls or adaptation to clients' restricted network connections, forexample.

A potential conferee 17(a) has navigated his or her WWW browser to Webserver 30(a), and has asked through the Web page presented to connect tothe meeting (as described above in the discussion of FIG. 2). There maybe alternative ways, indicated here as 30(b),(c), to connect to themeeting, including direct access to the meeting manager or its database34 (called here “Meeting DB”). The meeting manager uses this database tohold information concerning the meeting (the database need not be on thesame computer as the meeting manager). This information was created whenthe person who set up the meeting requested that the meeting bescheduled, gave descriptive information for the meeting, specified thekeys and privileges, and provided other administrative information. Thedatabase is reconfigurable and easily extensible to include many andvaried meeting attributes. It may be accessed by a programminginterface. Potential new conferee client 17(a) sends a request to jointhe meeting, and then supplies the key for the meeting that thepotential conferee has obtained previously. Potential client 17(a) mayalso send previously selected identification information such as icon,gong, etc., and this may be stored in Meeting DB or in some other sortof directory service. After the meeting manager has validated potentialclient 17(a), it sends a message that causes the client software to runon the potential client and then sends that client software the addressinformation for the CSS, such as a URL and port number. At that time,the client software may also receive address information for backup CSSsin case the connection to the meeting fails and automatic or manualattempts to reconnect to the initial CSS fail as well. The client thenconnects to the meeting, and may pass to the CSS its identificationinformation.

A CSS is created to supervise a single meeting. Themonitoring-filtering-queueing structures and procedures of FIGS. 8A,Bare performed by the CSS, so FIGS. 8A,B could be viewed as part of theinternal working of each CSS in FIGS. 11-22 (in the case of distributedserver functions described in FIGS. 9D-F, only part of FIGS. 8A,B mightbe descriptive of a particular CSS). Indeed, there will be a version ofFIG. 8A applying to each data stream the CSS handles as multipointreal-time traffic from a presenter client. The structure of FIG. 8Bshows schematically how these and other multiple input and output datastreams are processed. The CSS also handles other input from and outputto clients, such as information about attendee and presenter clientsthat helps with flow control, commands or requests from clients, labeledpointer icon positions, and other stream data and control traffic.

In FIG. 11, a dot-and-dash line 14 has been drawn around the structurethat corresponds to the term “server” in the earlier parts of thedescription of the present invention; this may be a helpful analogy, butthe description of the example here is only one possible explication ofserver functions.

FIG. 12 shows a slightly more complex situation than FIG. 11. Here, theserver manager has created three CSSs to supervise three meetings.Conferee client 17(a) (labeled here “Jim”) is simultaneously connectedto two meetings. If Jim is permitted, he can share the information hereceives from one meeting with the participants in the other.

Server managers are responsible for measuring network connectionbandwidth, reliability, CPU load, and other parameters, and determiningthe configuration of any and all CSSs they may own at any given timebased on these measurements and other considerations.

FIG. 13 shows a more complex arrangement than FIG. 12. Now, there aretwo CPUs, 38(a) and 38(b); each has its own server manager, but both aredirected by the same meeting manager. Server manager #1 36(a) hascreated two CSSs to handle two meetings, and server manager #2 36(b) hascreated a single CSS. Conferee client 17(a) is now connected to twomeetings on two different CPUs, possibly distant from each other.

FIG. 14 exemplifies the situation when there are several meetingmanagers by showing two meeting managers 32(a), (b) active. Each has itsown meeting database 34(a),(b). Each directs the server manager36(a),(b) on a single CPU 38(a,b). Server manager 36(a) has created twoCSSs 40(a),(b) to handle two meetings. The other server manager 36(b)has created a single CSS 40(c). The only connection pictured betweenthese two instances of the system is the presence of client 17(a) “Jim”in two meetings, one in each instance of the system.

If a need arises to let a second meeting manager set up meetings on aCPU that already has a server manager managed by a meeting manager, thenthe currently running server manager forks or clones itself and the newserver manager becomes the agent for the newly involved meeting manager.Thus, there is a one-to-one correspondence between server managersrunning on a given CPU and meeting managers that set up meetings on thatCpu. As an illustration, in Panel 1 of FIG. 15, meeting manager 32(b)sends a message to server manager 36(a) on CPU 38 that causes servermanager 36(b) to be created. Then afterward, in Panel 2, meeting manager32(b) and server manager 36(b) start the meeting through the new CSS40(b). Conferee “Jim” 17(a) has joined both meetings, but there need beno other relationship between the two meetings shown. In the situationsbelow, there will be no difference if server managers for differentmeeting managers are on the same CPU or on different CPUs.

FIG. 16 shows a single server manager 36 that has created three CSSs40(a)-(c) on one CPU 38, but now these CSSs all handle the same meeting(called “Meeting #3 ‘Sales’”). The same meeting might require additionalCSSs on the same CPU if process limitations were exceeded by a greatnumber of client connections and their requirements, or the like. Inorder to coordinate their work, the three CSSs communicate using asystem-provided server-server protocol. This protocol may use the samesort of blend of networking protocols as described for the client-serverconnection, or it may be quite different. For diagrammatic purposes,FIGS. 16-20 show interprocess communication links only between nearestneighbors; this is not meant to indicate that there are not otherdirectly established links, say between CSSs 40(a) and 40(c) in FIG. 16.This interprocess communication allows one CSS to send presenter clientoutput to another, for example; this was described above in thediscussion of FIG. 8A. More detail on this is given below in thediscussion of FIGS. 21 and 22. Conferee client 17(a) “Jim” is known totwo CSSs 40(b),(c), but is actively connected only to CSS #3 40(c). Theconnection to CSS #2 40(b) is a backup assignment (as described above inthe discussion of FIG. 9C). Should CSS #3 fail, then “Jim” canautomatically be connected to the meeting through CSS #2.

FIG. 17 shows a single meeting manager that directs two server managers36(a,b) that have created three CSSs 40(a,b,c) on two CPUs 38(a,b), andthese CSSs all handle the same meeting (called “Meeting #3 ‘Sales’”).The same meeting might require additional CSSs on additional CPUs if theCPUs were distant from each other, but closer to their respective setsof connected clients, or if process limitations were exceeded by a greatnumber of client connections and their requirements, or the like. Theadvantages described in the discussion of FIGS. 9A-G, 10A, B providenumerous reasons for multiple servers (which in this context meansmultiple CSSs) that handle the same meeting and that may be distributedover a number of CPUs. As in FIG. 16, the three CSSs communicate using asystem-provided server-server protocol. In addition, the two servermanagers communicate using this or a similar protocol in order tocoordinate the full span of this meeting. They exchange the performancemeasurements they make in order to adaptively configure the CSSs. Forexample, conferee client 17(a) “Jim” begins the meeting connected to CSS#2. At some point, the server managers determine that it is likely thatbetter service will be obtained by “migrating” this connection from CSS#2 40(b) to CSS #3 40(c) and so from CPU 38(a) to CPU 38(b) (asdescribed above in the discussion of FIG. 9B). This reconnection may beperformed automatically. Moreover, the creation of CSS #3 to handle thismeeting and the interprocess communication channels between CSSs andbetween server managers may be established automatically to improveperformance and balance loads (as further described above in thediscussion of FIG. 9B). The creation of additional CSSs, on the samemachine or on different machines, to handle the same meeting is thebasis of the scalability of the present invention.

It is possible for several meeting managers to each direct a servermanager that has created one or more CSSs, and these CSSs all handle thesame meeting. This is pictured in FIG. 18 with two meeting managers32(a),(b) and their respective databases 34(a),(b). This could be thesituation if two different companies own the meeting managers and mightwish to hold a joint meeting (here called “Meeting #4 ‘Joint Sales’”).The user or administrator that sets up such a multiply managed meetingmay manually declare the organization of the management (e.g., themeeting managers involved, the CPUs, the number of CSSs, and otherstructural, organizational, managerial parameters), or the setup may bedone automatically by the system, or it may be done interactively withautomated support. Here meeting managers 32(a),(b) also communicateusing the system-provided server-server protocol. They exchange themeeting information, so that their two databases 34(a),(b) areconsistent. They also exchange messages that indicate that this is to bea joint meeting, and they inform their server managers of this.Potential conferees join through either meeting manager. The appearanceof the meeting may be the same for all conferees, independent of thenumber of CSSs, server managers, CPUs, or meeting managers involved.Again, conferee client 17(a) “Jim” may be either backed up or migratedto another CSS (from 40(c) to 40(b)), even when the new CSS is on a CPUin the domain of a different meeting manager (from CPU 38(b) in thedomain of meeting manager 32(b) to CPU 38(a) in the domain of meetingmanager 32(a)).

FIG. 19 extends the situation of FIG. 18. The two meeting managers areshown directing their server managers in a joint meeting. This time,server manager 36(c) has created two CSSs 40(c),(d) for the meeting onthe same CPU 38(c), and meeting manager 32(a) has directed two servermanagers 36(a),(b) to create CSSs 40(a),(b) for the meeting on twodifferent CPUs 38(a),(b). Thus this is a combination of the situationspictured in FIGS. 16-18. Again, for diagrammatic purposes, FIGS. 19 and20 show interprocess communication links among server managers onlybetween nearest neighbors; this is not meant to indicate that there arenot other directly established links, say between server managers 36(a)and 36(c) in FIG. 19. As before, conferee client 17(a) “Jim” may beeither backed up or migrated to another CSS 40(b), even when the new CSSis in the domain of a different meeting manager.

If a CSS fails, the server manager process may create a new CSS, theclients may be migrated to other CSSs already active on the same CPU, orclients may be migrated to other CSSs newly created or already handlingthe meeting at other locations in the system, as indicated above. If aCPU fails or becomes isolated from the network or system, the meetingmanager process can attempt to reestablish network connection with theCPU and send the server manager information to create a CSS to handlethe meeting, or the meeting manager can communicate with one or moreserver managers on still accessible CPUs to create one or more CSSs tohandle the meeting. In addition, clients that have backup connectionsbeyond the failed CPU can be migrated to CSSs handling the meeting. Ifthere are several meeting managers participating in the management of ameeting, and a meeting manager becomes disabled or isolated from thenetwork, then another meeting manger may attempt to establishsufficiently many CSSs through server managers it can reach to carry theworkload. These adaptations may occur automatically, or be initiated bya system administrator.

The diagrams in FIGS. 11-19 suggest the range of variability incoordinating meetings and server functions. Many other combinations canbe formed from the situations pictured there or may be suggested bythem. For example, if a global directory service for meetings isprovided, then there could even be a layer of management above themeeting managers, which might be termed a Global Manager.

FIG. 20 illustrates a method for determining which CSSs pass outputinformation to other CSSs that are handling the same meeting. Theconfiguration of meeting managers, server managers and CSSs is the sameas in FIG. 19, except that one client 12 is presenting, the others 18are attending, and no backup link is shown. From time to time, theserver managers determine and agree on appropriate propagationtopologies that resemble a tree or trees (trees and their advantageswere described above in the discussion of FIGS. 10 A, B) and post a copy42 of the current choice at each of their CSSs. These are directedacyclic graphs (“DAGs”), which contain no loops, so that the CSS cansend out the information with no risk of endlessly cycling it. Eachstream being handled has its own propagation DAG, and they may changeindependently among streams; as described below, each stream may haveseveral propagation DAGs. In FIG. 20, presenter client 12 is viewed asthe root of the tree, and CSS #1 delivers presenter output informationto CSS #2 and to CSS #3; in turn, CSS #3 delivers that information toCSS #4; all CSSs also deliver the information to their connectedattendee clients 18.

FIG. 21 represents the situation of FIG. 20 with the topologicalinformation of the directed graph 42 emphasized by rearranging the CSSsto resemble the tree specified in the propagation DAG 42 of FIG. 20.

FIG. 22 represents the situation of FIG. 20 with the same topologicalinformation 42(a) emphasized in FIG. 21 and with the addition of anotherpossible propagation topology 42(b) for the same stream. Secondary andmultiple propagation DAGs for the same data stream allow the system toreroute the information being sent or to send it redundantly (both asdescribed above in the discussion of FIG. 9G).

The foregoing suggests that minimal server platform needs for thisexample would be a network connection and an operating system providingan interrupt service and multitasking, with or without hardware support.

System Extensions and Extendability

Up to this point, there have been multiple servers dealing with multipleclients, where “client” has referred to an enduser's computer or aninstance of the conferencing software running on it. But it may happenthat the reconfigurations, transformations, and routings performed by aserver or servers described above extend to assigning tasks tointermediate devices such as routers, bridges, gateways, modems,hardware codecs, and the like. There may be also be cases where a clientlacks display capabilities, but provides other functions to the systemor acts as a monitor or recorder of activity. There may beconfigurations where all server functions are performed on clientcomputers, so there is no specified server node in the networked system.In the preferred embodiment, performance may require that there be oneCSS per cpu.

Both the server and client software architectures are adaptable. Notonly may features be added on that increase the variety of communicationpossible, such as text chat, audio, and shared drawing areas, butproprietary codecs, transformations, stream operators, and the like maybe incorporated as plug-in modules. Addition of streams of abstract datatypes can be accommodated by the system and in turn allow the system tobe expanded and reconfigured.

When operating with data streams that admit asynchronous unnotifiedupdating (where intermediate updates can be dropped if they areobsoleted by later arriving data updates), which are those that areappropriate for multispeeding, the system is robust under occasionalloss of information and can thus take advantage of high-speed networkingprotocols like User Datagram Protocol (“UDP”) that do not providereliable transmission but provide greater network throughput performancethan reliable protocols. The system can also be configured to providesecure transport for a data stream, and so could be used to carry thedisplay command streams on which are based other image-sharing systems,but then there would be advantage from multispeeding, although thescalability and other advantages of the present invention would beavailable.

Codecs, transformations, and transcodings described above for theimagesharing example may have analogues that play similar roles withsimilar advantages in the system's handling of other data streams. It isalso recognized that there are other cases of transcoding from one typeof data to another, such as text to page image via rasterization, pageimage to text via optical character recognition, text to speech viaspeech synthesis, and speech to text via speech recognition. Such tasksmay involve transformations in different orders than described above,including decompression at the presenter client and compression at theattendee client. These possibilities are accommodated by the presentinvention.

The system contains no obstructions to operating across firewalls withpermissions. The system is compatible with agents, network proxies, andother stand-in entities, with a variety of network context and contentfilters, with multiple network protocols for client connections, withbandwidth matching transcoders, with hybrid wireless and landbasednetworks, with assymetric networks, with clustered CPU networks, withstreaming multimedia and signal processing systems, with serial,parallel, and vector processors, and with many other specializedtechnologies; properly configured and supplied with appropriatepermissions, the system either operates transparently with them orenjoys increased performance by employing and extending theiradvantages. Faster processor speed and greater bandwidth do not obviatethe utility of the present invention; instead, they improve itsperformance.

As mentioned before, the described communications system not onlyapplies to the transport of streams of image data, and to other examplesmentioned, but also generally applies to the transport, storing,replaying, scheduling, multispeeding, and other handling and processingof other data streams as well.

One way of seeing the flexibility of the system is to refer to FIG. 23,where several applications covering different separations in time andspace for the communicants are listed.

The above description is illustrative and not restrictive. Manyvariations of the invention will become apparent to those of skill inthe art upon review of this disclosure. The scope of the inventionshould, therefore, be determined not with reference to the abovedescription, but instead should be determined with reference to theappended claims along with their full scope of equivalents.

What is claimed is:
 1. A method for capturing information forpresentation in a conference, the method comprising: receiving a userinput at an interface of a first computer, the user input including aselection of at least one first area of a first display and a selectionat least one second area of a second display, wherein the first displayand the second display are associated with a first computer; andexecuting instructions at a processor, wherein the execution ofinstructions by the processor: translates the received user selection toboundary positions around the at least one first area and the at leastone second area, wherein the boundary positions are independent ofboundaries of the first display and boundaries of the second display,and selects a first frame image within the boundary positions around theat least one first area and a second frame image within the boundarypositions around the at least one second area.
 2. The method of claim 1,wherein the first display and the second display are respective firstand second monitors associated with the first computer.
 3. The method ofclaim 1, wherein the selection of the at least one first area of thefirst display is made independent of the selection of the at least onesecond area of the second display.
 4. The method of claim 1, wherein theselection of the at least one first area of the first display is made incoordination with the selection of the at least one second area of thesecond display.
 5. The method of claim 4, wherein the at least one firstarea and the at least one second area form at least one coordinatedcapture region.
 6. The method of claim 1, wherein the at least one firstarea and the at least one second area are shown in the respectivedisplays as respective first and second capture regions.
 7. The methodof claim 6, wherein the first and second capture regions are marked onthe respective first and second displays with graphical objects defininga shape of the respective first or second capture region.
 8. The methodof claim 1, wherein the boundary positions are independent of boundariesof applications shown on the first display, second display, or both. 9.The method of claim 1, wherein the at least one first area includes aplurality of first areas, and wherein the plurality of first areascorrespond to a plurality of layers shown in the first display.
 10. Themethod of claim 1, wherein the at least one second area includes aplurality of second areas, and wherein the plurality of second areascorrespond to a plurality of layers shown in the second display.
 11. Themethod of claim 1, wherein the boundary positions are automaticallyadjusted when the respective at least one first area or at least onesecond area is moved or resized in response to user input.
 12. Themethod of claim 1, wherein the execution of the instructions by theprocessor further transmits the selected first frame image and theselected second frame image to a second computer in a conferencingsystem.
 13. The method of claim 12, wherein execution of instructions bythe processor further: transcodes selected frame image data from a firstform associated with the first computer to a second form associated withthe second computer, wherein the selected frame image data is associatedwith the first frame image and the second frame image.
 14. The method ofclaim 13, wherein execution of instructions by the processor further:transmits the selected frame image data in the second form to the secondcomputer, wherein the first computer and the second computer areassociated with respective attendee clients of a conference.
 15. Themethod of claim 14, wherein the second form is a device-independentform.
 16. A non-transitory computer-readable storage medium, havingembodied thereon a program executable by a processor to perform a methodfor capturing information for presentation in a conference, the methodcomprising: receiving user input, the user input including a selectionof at least one first area of a first display and at least one secondarea of a second display, wherein the first display and the seconddisplay are associated with a first computer; translating the receiveduser selection to boundary positions around the at least one first areaand the at least one second area, wherein the boundary positions areindependent of boundaries of the first display and boundaries of thesecond display; and selecting a first frame image within the boundarypositions around the at least one first area and the second frame imagewithin the boundary positions around the at least one second area.
 17. Asystem for capturing information for presentation in a conference, thesystem comprising: an interface for receiving user input, the user inputincluding a selection of at least one first area of a first display anda selection at least one second area of a second display, wherein thefirst display and the second display are associated with a firstcomputer; a memory; a processor for executing instructions stored inmemory, wherein the execution of the instructions: translates thereceived user selection to boundary positions around the at least onefirst area and the at least one second area, wherein the boundarypositions are independent of boundaries of the first display andboundaries of the second display, and selects a first frame image withinthe boundary positions around the at least one first area and the secondframe image within the boundary positions around the at least one secondarea.